Back

The authors of this paper introduce a new method called prefix-tuning, which is a lightweight alternative to fine-tuning for natural language generation tasks. Fine-tuning is the standard approach for using large pretrained language models for downstream tasks, but it requires updating and storing all the parameters of the model, which can be computationally expensive and memory-intensive. In contrast, prefix-tuning keeps the language model parameters frozen and only optimizes a small continuous task-specific vector called the prefix.

Prefix-tuning draws inspiration from prompting, where the model is given an instruction or example before generating the output. In prefix-tuning, the prefix is a sequence of continuous task-specific vectors that is prepended to the input, allowing the subsequent tokens to attend to the prefix as if it were “virtual tokens”. The prefix is optimized during training and can be easily modified for different tasks, making prefix-tuning modular and efficient in terms of storage.

The authors apply prefix-tuning to two specific tasks: table-to-text generation and abstractive summarization. They use the GPT-2 model for table-to-text generation and the BART model for summarization. Their experiments show that by learning only 0.1% of the parameters, prefix-tuning achieves comparable performance to fine-tuning in full data settings and outperforms fine-tuning in low-data settings. Prefix-tuning also performs well on examples with topics unseen during training, demonstrating its ability to extrapolate to new examples.

The paper also explores other aspects of prefix-tuning, such as the impact of prefix length, the comparison with embedding-only ablation and infix-tuning, and the initialization of the prefix. They find that longer prefixes improve performance up to a certain threshold, tuning only the embedding layer is not as effective as full prefix-tuning, and initializing the prefix with activations of real words leads to better performance.

The authors discuss the implications of prefix-tuning for personalization and batching across users. They highlight that prefix-tuning allows for efficient personalization of language models for individual users, while still allowing batching of different user queries. They also discuss the inductive bias of prefix-tuning, suggesting that preserving the pretrained language model parameters may contribute to better generalization to unseen domains.

In conclusion, prefix-tuning is introduced as a lightweight alternative to fine-tuning for natural language generation tasks. It achieves comparable performance to fine-tuning with significantly fewer parameters, making it more computationally efficient and memory-friendly. The experimental results on table-to-text generation and summarization tasks show the effectiveness of prefix-tuning in different settings.

Words: 396